3D reconstruction is not just a low-level task: retrospect and survey

Although an image is a 2D array, we live in a 3D world. The desire to recover the 3D structure of the world from 2D images is the key that distinguished computer vision from the already existing field of image processing 50 years ago. For the past two decades, the dominant research focus for 3D reconstruction is in obtaining more accurate depth maps or 3D point clouds. However, even when a robot has a depth map, it still cannot manipulate an object, because there is no high-level representation of the 3D world. Essentially, 3D reconstruction is not just a low-level task. Obtaining a depth map to capture a distance at each pixel is analogous to inventing a digital camera to capture the color value at each pixel. The gap between low-level depth measurements and high-level shape understanding is just as large as the gap between pixel colors and high-level semantic perception. Moving forward, we would like to argue that we need a higher-level intelligence for 3D reconstruction. We would like to draw attention of the 3D reconstruction research community to put greater emphasis on mid-level and high-level 3D understanding, instead of exclusively focus on improving of low-level reconstruction accuracy, as is the current situation. In this report, we retrospect the history and analyze some recent efforts in the community, to argue that a new era to study 3D reconstruction at higher level is starting to come.